CLOSURES OF i^T-ORBITS IN THE FLAG VARIETY FOR GL{2n) 
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*isj ' Abstract. We characterize the 02n-orbits in the flag variety for GL2n with 

rationally smooth closure via a graph-theoretic criterion. We also give a nec- 
OJJ, essary pattern avoidance criterion for rational smoothness and conjecture its 

3 ' sufficiency. 
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1. Introduction 



p^ ■ Let G be a complex reductive group with Borel subgroup B and let K = G^ 

^ \ be the fixed point subgroup of an involution of G. In this paper we develop the 

program begun in ^M09 and continued in |MT091 IMIO] , seeking to characterize the 
iiT-orbits in G / B with rationally smooth closure via a combinatorial criterion. Here 
C^ . we treat the case G — GL{2n, C),K = 0{2n, C) (and prove some results about the 

general case G = GL{m,C),K = 0(m,C)). We focus on the order ideal of orbits 
with closures contained in a fixed one O; this is an interval inside the poset of all K- 
orbits in G/B, ordered by containment of closures. Richardson and Springer have 
^ . defined an action of the braid monoid corresponding to the Weyl group W oi G on 

■^ I this poset [RS90| . which we use to make it and its order ideals into graphs. Our 

OO . criterion for rational smoothness of O is given in terms of the degree of the bottom 

^^ ' vertex of this order ideal; it is known that this degree condition is necessary for 

rational smoothness in general jBr99) . We also give a necessary pattern avoidance 
r^ ' criterion for rational smoothness (for any GL(m, C)) and conjecture its sufficiency. 

2. Preliminaries 



Set G = GL{m,C),K = 0{m,C). Let B be the subgroup of upper triangular 
matrices in G. The quotient G/B may be identified with the variety of complete 
flags Vb C Vi C • • • C Vm in C™ . The group K acts on this variety with finitely 
many orbits; these are parametrized by the set /,„ of involutions in the symmetric 
group 5'.^ |M088| IRS90J . In more detail, let (•, •) be the standard symmetric bihnear 
form on C™, with isometry group K. Then a flag Vb C • • • C Vm lies in the orbit 
Ott corresponding to the involution tt if and only if the rank of (■,■) on Vi x Vj 
equals the cardinality #{fc : 1 < A: < i,7r{k) < j} for all I < i,j < m. 

We use the same definition of pattern avoidance for permutations as in [MlOj . 
decreeing that tt = tti . . . nm (in one-line notation) includes the pattern ^ = fii . . . fj,r 
if there are indices ii < 12 < ■ ■ ■ < ir permuted by tt such that tt^ . > tt^^ if and only 
if ^Lj > /ifc. We say that tt avoids /i if it does not include it. 

There are well-known poset- and graph-theoretic criteria for rational smoothness 
of complex Schubert varieties due to Carrell and Peterson. The poset criterion does 
not extend to our setting but the graph one does. To state it we first recall that 
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the partial order on /,„ corresponding to inclusion of orbit closures is the reverse 
Bruhat order jRS90| . Then Im is graded via the rank function 

r(7r) = [m^/4j - ^ {n{i) - i - #{fc £ N : i < fc < 7r(i), 7r(fc) < i]) 

i<Tr{i) 

where [to^/4J denotes the greatest integer to rn^/A and ^(Tr) equals the difference 
in dimension between O^ and Oc, the unique closed orbit, corresponding to the 
involution wq — m . . .1 [RS90J . Let /^ be the interval consisting of all tt' < tt in 
the reverse Bruhat order. We make /^ into a graph by decreeing that the vertices 
fj, and v in it are adjacent if and only if either v = tfit ^ ^ for some transposition t 
in Sm, OT v — tfi for some transposition t in Sm with tfit — ii and m is even; write 
v = t- fi ii either of these conditions holds. Then a necessary condition for Ott to be 
rationally smooth is that the degree of wq in /^r must be ^(Tr); in fact, all vertices 
'w~^wqw conjugate to wq in W and lying in I-^ must have this degree |Br99|, 2.5]. 

3. Main result 

Our first result is a pattern avoidance criterion valid for all m. 

Theorem 1. With notation as above, the orbit O^ has rationally singular clo- 
sure whenever tt contains one of the twenty-four had patterns 14325, 426153, 154326, 
124356, 153624, 351426, 213654, 321465, 3614725, 1324657, 2137654, 4321576, 5276143, 
5472163, 2135467, 1243576, 1657324, 4651327, 57681324, 65872143, 13247856, 34125768, 
341258967, 749258163. The same holds if it contains the pattern 2143, provided that 
there are an even number of fixed indices of it between 21 and 43 (e.g. tt — 21354687, 
where the 43 occurs in the last two indices of tt). 

Proof. One checks first that the degree of wq in /^ is greater than r(7r) for any 
TT in the list above, except for 2137654 and 4321576. The closures of the orbits 
corresponding to these two permutations are also rationally singular, as can be 
shown by computing their intersection cohomology (via Kazhdan-Lusztig-Vogan 
polynomials) directly (e.g. via ATLAS, available at http://www.liegroups.org). If 
TT is obtained from one of the bad patterns above other than 2143 by adding one 
or more fixed points, then one computes that the degree of wq is again more than 
r(7r), except for 2134765, 3214576, 2137564, and 4231576; these cases may again be 
checked directly via ATLAS, and any involution tt' obtained from them by adding 
one fixed point has the degree of wq bigger than r(7r'). The same holds if tt is 
obtained from 2143 by adding two or more fixed points, with an even number of 
them lying between 21 and 43. If tt is obtained from 2143 by adding just one fixed 
point not lying between 21 and 43, then the unique vertex conjugate to wq having 
this fixed point has degree larger than r(7r). Finally, if tt is obtained from one of 
the bad patterns by adding fixed points as above and then pairs of flipped indices, 
then one argues as in the proof of the Lemma in |M10j that some vertex in I^^ has 
degree greater than r(7r). Hence in all cases Oj^ is rationally singular. D 

Along these lines we also have the following 

Conjecture. The conditions of Theorem 1 are sufficient for rational smoothness of 
Ott', in addition, this orbit closure is smooth if and only if tt avoids the twenty-four 
bad patterns above and 2143 (without qualification) and 1324. 

We specialize to the even case m = 2n in our main result. 
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Theorem 2. Assume that m ~ 2n is even. The orbit O-n has rationally smooth 
closure if and only if the degree of wq in I^ is r(7r). 

Proof. Set O — O-n . We have already noted that the degree condition is necessary, 
so suppose that it is satisfied. We begin by constructing a shce of O to Oc at a 



particular flag, as follows. Fix a basis (e^) of C " such that {ci 



1 if i + j 



2n + 1 and (e^, e^) — otherwise, where as above (•, •) is the symmetric form. Let 
{aij) be a family of complex parameters indexed by ordered pairs («,j) satisfying 
either i<n<j or n<i<j. We assume that a^ 



but otherwise put no restrictions on the a^ 



a2n+l-j,2n+l-i lH < U < j 

Define a basis (bi) of C^" via 



2n 



Ci + y Qij Cj \i i < n 



2n 

■j=i+i 



otherwise 



Then the Gram matrix G :— (gij — (bi, bj) of the bi relative to the form satisfies 

9ji 



9^j 



ii i < j < n 

ii j < i < n 
aj,2n+i-i iii<n<j<2n+l~i 
1 ii i < n < j = 2ji + 1 — i 

9ji ii j <n <i 

otherwise 

Thus the matrix G is symmetric and has zeroes below the antidiagonal from lower 
left to upper right. The antidiagonal entries are all 1. Now one checks that the set 
S' of all fiags Vo C . . . C V2n where (bi) runs through all bases obtained as above 
from the Uij and Vi is the span oi bi, . . .bi is a sHce of G/B to Oc at the flag fc 
corresponding to the basis (e^), which in turn corresponds to the point P where 
all aij = |Br99[ 2.1]. Intersecting S' with O we get a sHce S oi O to Oc at P, 
defined by the vanishing of certain minors in the Gram matrix G. It is known that 
S is rationally smooth (resp. smooth) at P if and only if it is rationally smooth 
(resp. smooth) everywhere, or if and only if O is rationally smooth (resp. smooth) 
everywhere |Br99[ 2.1]. This construction works with minor modifications for odd 
m as well. 

We now show that S is rationally smooth at P by verifying the conditions of 
[Br99l 1.4]. Define an action of the n-torus T = T" on the matrix G by multiplying 
the first n rows and columns by ii, . . . , i„, respectively, while multiplying the last n 
rows and columns by i~^, . . . , t^^, respectively; this action preserves the Is on the 
antidiagonal and the vanishing of the minors that define the slice S. (T is just a 
maximal torus of K.) Then the weights of T occurring in the tangent space at P of 
the big slice S' are those of the form 2ei, e^ + Cj, or e^ ~ ej for some 1 < i < j < n 
and all occur with multiplicity one; each corresponds also to a variable in S and to a 
transposition t in S2n such that the vertices t-wg are distinct. In particular, they all 
lie on one side of a hyperplane and P is an attractive fixed point of both S' and S. 
The subtori T' of T of codimension one such that the fixed point subvariety (S')'^ 
of the big slice S' under the T'-action contains more than one point correspond 
exactly to the above weights. Now the hypothesis on the degree of wq implies that 
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that there are [?n^/4j — r(7r) distinct conjugates c — t ■ wo oi wo hy a transposition 
t with c ^ TT, where t corresponds to one of the above weights. Writing c as 
Ci . . . C2n in one-hne notation, let i be the smallest index such that if tti . . . tt^ is 
rearranged in increasing order as Tr[ . . .n'^ and similarly ci . . . Ci is rearranged as 
c']^ . . . c^, then tt' > c' for some j. Equating the minor of G corresponding to the 
first i rows and columns ci, . . . , c^ to 0, we get a polynomial in the variables of S in 
which exactly one of the monomial terms is either the first or second power of the 
variable corresponding to (the weight of) t. Hence the dimension of the fixed point 
subvariety iS"^ is 1 if T' corresponds to a i not arising from one of the conjugates 
c and otherwise, whence Brion's conditions [Br99t 1.4(ii),(iii)] are satisfied. 

Now we must verify condition [Br99| 1.4(i)]. By hypothesis there are c vertices 
V = twot or V = two adjacent to wq in /^, where c is the codimension of O in G / B. 
Write each such v dsvi . . . z;2n and let i be the smallest index with tti . . . tt^ ^ vi . . .Vi 
in the standard (entry by entry) partial order on sequences used to characterize 
Bruhat order [P82j . Thus if tti . . . tt^ is rearranged in increasing order as n[ . . .tt'^ 
and similarly vi . . .Vi is rearranged as v'l . . .v[, then tt' > u' for some j. Choose the 
least such j and let ri, . . . rj be the positions of v'l, . . . u' among vi . . .Vi. Then S is 
defined by the vanishing of the minor M^, of G of the submatrix Sy corresponding 
to columns v[, . . . ,v'^ and rows ri, . . .Vj, for all v. All but at most two rows of Sy 
contain a 1 and similarly for the columns. Use row and column operations to zero 
out all the entries other than 1 in any row or column of Sy with a 1, obtaining 
a new matrix Sy with the same determinant My. Then requiring that all My 
vanish amounts to imposing a set of linear and quadratic conditions on a new set 
of variables, consisting of the old variables Gij either not occurring in any Sy or 
originally occurring in a row or column with a 1 in some 5*1,, together with the 
entries pij of the new matrices Sy not lying in the same row or column as a 1 . Thus 

5 may be realized as the product of a suitable affine space C and a variety with 
only one singular point (at the origin) and it satisfies [Br99t l-4(i)], as desired. 

D 

We have already observed that Theorem 2 fails for odd m; indeed, the orbit 
closure O^r indexed by tt = 2137654 in GL(7, C)/B is rationally singular and yet /^r 
satisfies the degree criterion at all vertices conjugate to the closed one. A smaller 
example has tt — 21435; here the degree criterion holds at the closed orbit 54321, 
but fails at 43215, so again Ott is rationally singular. 
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